Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-6887][VL] Daily Update Velox Version (2024_08_16) #6872

Merged
merged 8 commits into from
Aug 16, 2024
Merged

[GLUTEN-6887][VL] Daily Update Velox Version (2024_08_16) #6872

merged 8 commits into from
Aug 16, 2024

Conversation

GlutenPerfBot
Copy link
Contributor

Upstream Velox's New Commits:

fe9605f74 by Jialiang Tan, Change SharedArbitrator extra config string based (10685)
9523840ac by xiaoxmeng, Add to support byte input stream back by file (10717)

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

jinchengchenghh
jinchengchenghh reviewed Aug 16, 2024
View reviewed changes
cpp/velox/shuffle/VeloxShuffleReader.cc
@@ -450,7 +451,7 @@ std::shared_ptr<ColumnarBatch> VeloxSortShuffleReaderDeserializer::deserializeTo
return std::make_shared<VeloxColumnarBatch>(std::move(rowVector));
}

class VeloxRssSortShuffleReaderDeserializer::VeloxInputStream : public facebook::velox::ByteInputStream {
class VeloxRssSortShuffleReaderDeserializer::VeloxInputStream : public facebook::velox::GlutenByteInputStream {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I make a workaround here because some API is removed by facebookincubator/velox#10717 and https://github.com/facebookincubator/velox/pull/10761/files

Can you help to refactor it when you are free? @marin-ma Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @zhli1142015

@jinchengchenghh
Copy link
Contributor

I meet this exception, because we call gluten::Runtime::registerFactory(gluten::kVeloxRuntimeKind, veloxRuntimeFactory); twice, create the velox backend in MemoryManagerTest and MultiMemoryManagerTest, do you have any suggestion? @zhztheplayer

/mnt/DP_disk1/code/gluten/cpp/build/velox/tests# ./velox_memory_test
Running main() from /mnt/DP_disk1/code/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/gtest-src/googletest/src/gtest_main.cc
[==========] Running 3 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 2 tests from MemoryManagerTest
[ RUN      ] MemoryManagerTest.memoryPoolWithBlockReseravtion
[       OK ] MemoryManagerTest.memoryPoolWithBlockReseravtion (0 ms)
[ RUN      ] MemoryManagerTest.memoryAllocatorWithBlockReservation
[       OK ] MemoryManagerTest.memoryAllocatorWithBlockReservation (0 ms)
[----------] 2 tests from MemoryManagerTest (0 ms total)

[----------] 1 test from MultiMemoryManagerTest
unknown file: Failure
C++ exception with description "Runtime factory already registered for velox" thrown in SetUpTestSuite().
[ RUN      ] MultiMemoryManagerTest.spill
/mnt/DP_disk1/code/gluten/cpp/velox/tests/MemoryManagerTest.cc:342: Skipped

[  SKIPPED ] MultiMemoryManagerTest.spill (0 ms)
[----------] 1 test from MultiMemoryManagerTest (0 ms total)

[----------] Global test environment tear-down
[==========] 3 tests from 2 test suites ran. (33 ms total)
[  PASSED  ] 2 tests.
[  SKIPPED ] 1 test, listed below:
[  SKIPPED ] MultiMemoryManagerTest.spill
[  FAILED  ] 0 tests, listed below:

 0 FAILED TESTS
[  FAILED  ] MultiMemoryManagerTest: SetUpTestSuite or TearDownTestSuite

 1 FAILED TEST SUITE

@zhouyuan
Copy link
Contributor

C++ exception with description "Runtime factory already registered for velox" thrown in SetUpTestSuite().

seems related with single-thread execution model.

@zhouyuan zhouyuan changed the title [VL] Daily Update Velox Version (2024_08_16) [GLUTEN-6887][VL] Daily Update Velox Version (2024_08_16) Aug 16, 2024
@github-actions GitHub Actions
Copy link

#6887

@jinchengchenghh jinchengchenghh merged commit 6dcf83f into apache:main Aug 16, 2024
44 checks passed
@GlutenPerfBot
Copy link
Contributor Author

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_08_16_2024_time.csv log/native_master_08_15_2024_1845f30c4c_time.csv difference percentage
q1 40.99 40.51 -0.474 98.84%
q2 30.15 30.39 0.234 100.78%
q3 53.31 53.23 -0.080 99.85%
q4 42.30 44.25 1.947 104.60%
q5 105.58 102.93 -2.651 97.49%
q6 12.97 12.09 -0.882 93.20%
q7 116.05 116.78 0.730 100.63%
q8 116.44 117.10 0.664 100.57%
q9 168.30 169.16 0.867 100.52%
q10 66.48 66.73 0.254 100.38%
q11 27.13 26.55 -0.582 97.86%
q12 29.28 30.12 0.844 102.88%
q13 52.14 51.22 -0.925 98.23%
q14 25.75 24.79 -0.959 96.27%
q15 54.10 54.28 0.175 100.32%
q16 18.35 18.18 -0.171 99.07%
q17 131.98 130.17 -1.804 98.63%
q18 201.13 196.71 -4.418 97.80%
q19 28.08 27.00 -1.075 96.17%
q20 43.76 47.52 3.756 108.58%
q21 383.76 376.64 -7.111 98.15%
q22 15.87 15.77 -0.096 99.40%
total 1763.87 1752.12 -11.757 99.33%

sharkdtu pushed a commit to sharkdtu/gluten that referenced this pull request Nov 11, 2024
@GlutenPerfBot
[GLUTEN-6887][VL] Daily Update Velox Version (2024_08_16) (apache#6872)
9528842
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@marin-ma marin-ma marin-ma left review comments

@jinchengchenghh jinchengchenghh jinchengchenghh approved these changes

Assignees
No one assigned
Labels
BUILD VELOX
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@GlutenPerfBot @jinchengchenghh @zhouyuan @marin-ma